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Multimedia messaging method and system 



The invention relates to a method of controlling retransmission of a content 
item contained in a multimedia message. The invention further relates to a system arranged 
for controlling retransmission of a content item contained in a multimedia message. 



Multimedia Messaging Service (MMS) is a standard that lets users send and 
receive multimedia messages with formatted text, graphics, audio and video clips. MMS 
messages can be sent to (MMS enabled) mobile phones or standard e-mail addresses. The 
sending entity can for instance be a mobile phone or an application running on a computer. 

10 Fig. 1 shows the generic MMS Person-to-Person flow. A first person wants to 

send an MMS message to a second person. He takes his mobile phone 101 or other MMS 
user agent, composes the message and includes one or more multimedia objects in the 
message. After hitting the "send" button, the multimedia message (text and the multimedia 
objects) is transmitted over a mobile network 1 10 to a Multimedia Messaging Service Center 

15 (MMSC) 120. 

The mobile network 1 10 could be e.g. GSM, GPRS, UMTS or a comparable 
system. As multimedia messages can potentially grow very big, a broadband mobile network 
is preferred. The first person could also have composed the message on his personal 
computer connected to the Internet. The message would then have been sent to the MMSC 
20 120 over the Internet. Other ways to transmit multimedia messages are of course also 

possible. It is important to note that the message is always sent to the MMSC 120 and not 
directly to the recipient. This architecture is an example of a store-and-forward messaging 
system. 

The multimedia message is received by the MMSC 120. From there it is 
25 forwarded to the intended recipient (the second person). The multimedia message is once 
more transmitted over a mobile network 130, this time to the mobile telephone or other user 
agent 141 of the second person. Upon arrival of the multimedia message, this mobile 
telephone processes the content and presents the message to the second person. The two 
mobile networks 110, 130 in question may or may not be the same. 
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In the above example, the first person used the second person's telephone 
number as the address to which the multimedia message was to be sent. He could also have 
used the second person's e-mail address. The multimedia message would then have been 
forwarded by the MMSC 120 to the second person's e-mail box, from which the second 
5 person could have read it using a standard e-mail client. 

Fig. 2 shows the generic MMS Application-to-Person flow. Multimedia 
messaging is of course not restricted to messages sent by persons to persons. Many 
automated services can be offered. For instance, a popular service is the making available of 
ringtones or logos for mobile telephones. In such an application, a user can send a request for 

10 a particular ringtone or logo to a server from his mobile telephone. An application 201 on the 
server retrieves the appropriate ringtone or logo and transmits it as a multimedia message to 
the MMSC 120. Once the message arrives at the MMSC 120, it is handled and forwarded in 
the same way as described above with reference to Fig. 1. When the mobile phone 141 
receives the multimedia message, it extracts the ringtone or logo. 

15 Delivering ringtones and logos is already very popular with the standard text- 

based Short Messaging Service (SMS). However, MMS has a much broader application. A 
potential use of Application-to-Person MMS is the provision of (high) value content. An 
example could be a service which can provide the latest music samples or video clips. The 
user pays an extra amount for that message to cover the content licensing costs, as most of 

20 this content is copyrighted or protected by other intellectual property rights. 

A problem occurs if both presented models are combined: The user who paid 
for the content (application-to-person) can forward this message (and thus the content) to 
another user (person-to-person), only paying for the transmission of the message, but not 
paying any license fees due for forwarding the content. 

25 

It is an object of the invention to be able to restrict or otherwise control the 
forwarding of multimedia content. 

This object is achieved according to the invention in a method as claimed in 
30 claim 1, and in a system as claimed in claim 10. According to the invention the service 

provider who originally provided the content item is contacted and forwarding is conditional 
upon his approval. This has the advantage that the entity handling the forwarding does not 
need to keep track of policy rules regarding when it is or is not allowed to forward a content 
item. Furthermore, any such rules can be changed at any time by the provider without having 
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to update anything in the forwarding system. The sytem simply contacts the provider and 

learns from the response whether forwarding is permitted. 

In an embodiment the invention uses robust fingerprints to identify the content 

transmitted in a multimedia message. The provider can be identified using a database lookup 
5 using the fingerprint as a key. A single fingerprint may be associated with identifiers for 

plural providers, which means that any of these providers can approve forwarding. This 

makes the invention very flexible in terms of licensing options and enables for example 

price-based competition between providers. 

Restricting forwarding, for example by disallowing retransmission of the 
10 entire message, or by allowing retransmission of the message but disallowing retransmission 

of the content item, is usually desired. Another option is to allow forwarding of the content 

item and tracking such forwarding. By tracking the content, useful information about the 

distribution of the content can be gathered (for marketing purposes), but also business rules 

can be added to limit this distribution of the content. 
1 5 Advantageous embodiments are set out in the dependent claims. 

These and other aspects of the invention will be apparent from and elucidated 
with reference to the embodiments shown in the drawing, in which: 
20 Fig. 1 schematically illustrates a multimedia messaging system in which 

messages from one user are passed via an MMSC to another user; 

Fig. 2 schematically illustrates a multimedia messaging system in which 
messages from an application are passed via the MMSC to a user; 

Fig. 3 schematically illustrates an preferred embodiment of the invention in 
25 which a fingerprinting module is added to the MMSC; 

Fig. 4 schematically illustrates an embodiment of the invention in which a 
fingerprint-based identification module is added to the MMSC; 

Fig. 5 schematically illustrates a Media Mediation System coupled to the 

MMSC; and 

30 Fig. 6 schematically illustrates an embodiment of the MMSC. 



Throughout the figures, same reference numerals indicate similar or 
corresponding features. Some of the features indicated in the drawings are typically 
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implemented in software, and as such represent software entities, such as software modules 
or objects. 

An illustration of a preferred embodiment of the invention is shown in Fig. 3. 
In this embodiment, the MMSC 120 is extended with a fingerprinting module 310 through 
5 which all content items present in multimedia messages must pass. The fingerprinting 

module 3 1 0 computes a fingerprint for the content item in question. This fingerprint serves as 
an identifier for the content item. 

One method for computing a robust fingerprint is described in international 
patent application WO 02/065782 (attorney docket PHNL0101 10), although of course any 
1 0 method for computing a fingerprint can be used. The fingerprinting module 310 then 

performs a database lookup in database 3 1 1 to determine whether this particular fingerprint is 
present in the database 3 1 1 . If not, the fingerprint of the content item in question is added to 
the database 311 together with an identifier for the sender of the content item. This sender 
then most likely is the original provider of the content item. If desired, an additional check 
1 5 can be performed to determine whether this is true before adding the sender to the database 
311. 

Another option is to add the fingerprint of any content item to the database 
3 1 1 when it is first made available by the provider, not when it is first transmitted from the 
provider to a client. 

20 The MMSC 120 is also provided with a fingerprint-based identification 

module 410, as shown in Fig. 4. The fingerprint-based identification module 410 retrieves 
additional information from the database 3 1 1 based on the fingerprint computed by the 
fingerprinting module 3 10. If additional information can be found, this indicates that the 
content has passed through an MMSC previously. This is proof that the content is being 

25 forwarded. As such forwarding is to be prevented or at least restricted, the MMSC 120 can 
now take appropriate action. It is noted that if multiple MMSCs are going to be used, the 
database 311 should be accessible to all MMSCs. 

The simplest action is to simply delete the multimedia message rather than 
forwarding it. However, this may not be acceptable to users. Usually one part of the message 

30 (such as a picture or ringtone) represents forwarded content. Other parts, such as 

accompanying text messages or pictures, are original content. The original content should not 
be blocked simply because there is also a forwarded part in the message. The MMSC 120 
could strip out this forwarded piece of content and pass on the remaining parts of the 
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multimedia message. Preferably, any parts that have been stripped out are replaced by a 
message indicating that they have been stripped out and why. 

Alternatively, a license fee required for forwarding the content could be 
computed. This license fee can then be charged to the sending user's account, if this 

5 information is available. This could work well e.g. with music content, as music rights 
clearinghouses are available to negotiate a general compensation or license fee. 

The MMSC 120 could also convert the content to a relatively low quality 
version before forwarding the content to the intended recipient. This way, the intended 
recipient would still get the message and the content, but because of the low quality it would 

10 not be very valuable. He would then be encouraged to download the original version from the 
MMS application 201 himself. 

The action can also comprise reporting the distribution of this content to a 
third party such as the service provider. Distribution can then be made conditional upon 
approval from the provider. This embodiment works best if an identifier for the provider is 

1 5 embedded in the content item. 

Another example is the allowance of the forwarding, but billing the sending 
user a premium price for it. In mobile networks, the sending user can be identified e.g. 
through the telephone number to allow such billing. Price information can be appended to the 
message or be supplied in a separate message. 

20 Appropriate action could be taken based on the retrieved additional 

information. For example, if the additional information identifies the initial recipient of the 
content being forwarded (i.e. the person who received it from the MMS application server in 
Fig. 2), the MMSC 120 could compare this identifier against an identifier for the sender of 
the intercepted multimedia message. If these two are the same, it could permit forwarding. 

25 This way, the initial recipient can forward content to his friends, but those friends cannot pass 
the message on to anyone else. 

Alternatively, the identifier could be used to track down the initial recipient of 
the content, so that he can be charged a certain amount of money for this premium service, or 
so that he can be blocked from using the MMS application. He could also be offered a 

30 subscription before he is allowed to forward content items. 

If the additional information identifies the MMSC which added the fingerprint 
to the database 311, or the operator of one or more MMSCs, the MMSC 120 could allow 
forwarding of the message if it were the same MMSC as the MMSC which added the 
fingerprint. This permits forwarding only to and from users of the same MMSC. 
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If the additional information identifies the provider of the content item, 
retransmission can be made conditional upon approval by that provider. This is preferably 
realized by having the fingerprinting module 310 store the computed fingerprint associated 
with an identifier for the provider in the database 3 1 1 if no identifier for the provider could 
5 be retrieved from the additional information. This way, the fingerprint-based identification 
module 410 can retrieve this identifier and contact the provider to obtain permission to 
retransmit the message and/or the content item, or to take any other action as explained 
above. 

If the additional information identifies plural providers, retransmission can be 

10 made conditional upon approval by any of the plural providers. The fingerprint-based 

identification module 410 can now contact all of these providers and permits retransmission 
if at least one of them approves. If more than one provider approves, the most liberal 
approval could be applied, or the sender could be asked to pick one of the providers, or a 
selection could be made in some other way. 

1 5 Identifying plural providers can be implemented by storing the fingerprint 

associated with one provider when it is found that the fingerprint is not yet in the database 
311. If subsequently the fingerprinting module 310 notices that the fingerprint is in the 
database 3 1 1 but not associated with the sender of the content item, then an identifier for this 
sender can be added to the entry for that particular fingerprint. The fingerprint-based 

20 identification module 410 is then able to retrieve all the providers of the content item. 

By using fingerprints, it becomes possible to identify one particular content 
item regardless of the transmission medium or the format in which it is encoded. For 
example, it can now be determined that two content items contain the same piece of music, 
even if the respective items originate from different providers. Using fingerprints is also 

25 robust against end users tampering with information such as filename, file size or any 
information embedded in the content item. 

The fingerprinting module 310 and the fingerprint- based identification module 
410 can be embodied as separate devices to be installed in the MMSC 120, or as separate 
devices connected to the MMSC 120. For example, the module 310 could be designed to 

30 operate as a proxy to be connected between MMS application 201 and MMSC 120, making 
operation transparent. Similarly, the module 410 can be designed as a proxy between MMSC 
120 and mobile network 130. The modules 310, 410 can be embodied as one single device or 
module. They can be embodied in whole or in part in hardware or in software. 
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The entity operating the MMSC 120 is able to optimize content at the MMSC 
120 for the receiving user agent using media mediation system 510, as shown schematically 
in Fig. 5. Optimization can be the adaptation or transcoding of the format (for images e.g., 
GIF to JPEG), the size (for video e.g., from CIF to QCIF), the bitrate (for audio e.g., 128kbps 
5 to 20kbps) and other parameters. Since this media mediation system 510 has access to, and 
knowledge about the multimedia content of the message, the modules 3 10, 410 are then 
preferably positioned in this system 510. 

An illustrative embodiment of the MMSC 120 is shown schematically in Fig. 
6. The functionality of the MMSC 120 can be divided conceptually into a server portion 601 
10 and a relay portion 65 1 . Briefly speaking, the server portion 601 provides the store-and- 
forward architecture and the relay portion 651 transcodes and delivers the messages. 

A message is received by the receiver component 610 of the server 601. The 
message is processed to detect how it should be handled and to whom it should be forwarded. 
In accordance with the invention, the contents of the message is processed by the fingerprint- 
1 5 based identification module 41 0 to detect whether the multimedia content of the message has 
already been forwarded. If such detection is positive, the MMSC 120 takes appropriate 
action, as explained above. 

If forwarding of the message is acceptable, there are several possible paths the 
message can take. Usually the message will be passed on to the relay 65 1. The relay 651 
20 comprises the fingerprinting module 310 and the media mediation system 510 which 

transcodes the message to a form suitable for handling by the recipient. The message is then 
passed on to the mobile network 130 so that it can be delivered to the intended recipient. 

The message can also be passed on to an SMTP interface 620 connected to a 
network 621 such as the Internet. This interface formats the multimedia message in 
25 accordance with Internet e-mail standards, and then forwards the message as an Internet e- 
mail message to the intended recipient. The SMTP interface 620 may or may not perform the 
above fingerprint-based check on the multimedia message before forwarding it. If the SMTP 
interface 620 does so, further forwarding of the resulting e-mail message to another recipient 
via the MMS network becomes impossible. However, as the Internet e-mail network is not 
30 very strictly regulated, it will be very difficult to enforce forwarding restrictions over the e- 
mail network itself. 

If the recipient is not on the same mobile network as the sender of the 
multimedia message, the relay cannot forward the message itself to the intended recipient In 
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' this case, the server passes the message on to an MMSC 630 which is connected to the 
correct mobile network. 

The fingerprint-based identification module 410 can also be positioned in the 
relay portion 651 of the MMSC 120. This has the advantage that the module 410 and module 
5 310 could e.g. be realized as a single device. 

The functionality of the transcoding module 510 can be extended to more than 
just converting between formats, sizes, bitrates etcetera. For instance, an interface with MMS 
applications could be provided. This way, an MMS application could for instance 
dynamically generate content specifically adapted for one particular mobile device. 

10 It should be noted that the above-mentioned embodiments illustrate rather than 

limit the invention, and that those skilled in the art will be able to design many alternative 
embodiments without departing from the scope of the appended claims. For example, the 
modules 310, 410 could be incorporated in the user agents 101,141. Instead of robust 
fingerprints, also for example cryptographic hashes, CRCs, filename-based identifiers or 

15 watermark-based identifiers could be used. A combination of different techniques for 
obtaining an identifier is also possible. 

The invention is of course applicable to any store-and-forward system, for 
example i-mode/i-mail, WAP and all future mobile premium content ways of transport, not 
just the MMS system described above. 

20 The content items that can be transported could be any type of content, for 

example (java) games, greeting cards (still and animated), location based services (eg city- 
maps, etc.), premium information content (news reports* pictures, stock-market info + 
graphics), screensavers, images/pictures (can be anything: music artists, playboy playmates, 
etc.), audio or video clips and so on. 

25 It is for instance also possible to implement the fingerprinting and fingerprint- 

based lookup functionality according to the invention in an Internet or Intranet e-mail server 
system. This way e-mail messages cannot be used to indefinitely forward multimedia content. 
Analysis of the contents of e-mail attachments is already done by anti-virus software, and a 
similar approach could be used to compute fingerprints in attachments. 

30 In the claims, any reference signs placed between parentheses shall not be 

construed as limiting the claim. The word "comprising" does not exclude the presence of 
elements or steps other than those listed in a claim. The word "a" or "an" preceding an 
element does not exclude the presence of a plurality of such elements. The invention can be 
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implemented by means of hardware comprising several distinct elements, and by means of a 
suitably programmed computer. 

In the device claim enumerating several means, several of these means can be 
embodied by one and the same item of hardware. The mere fact that certain measures are 
5 recited in mutually different dependent claims does not indicate that a combination of these 
measures cannot be used to advantage. 



